#6: Implemented DataHandler class and extract/remove/namelist #7
Conversation
|
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7 +/- ##
=======================================
Coverage ? 78.15%
=======================================
Files ? 22
Lines ? 1927
Branches ? 0
=======================================
Hits ? 1506
Misses ? 421
Partials ? 0 ☔ View full report in Codecov by Sentry. |
ifsbench/data/renamehandler.py
Outdated
| mode: `RenameHandler.RenameMode` | ||
| Specifies how the renaming is done (copy, move, symlink). | ||
|
|
||
| mode: `RenameHandler.RenameMode` |
There was a problem hiding this comment.
| mode: `RenameHandler.RenameMode` | |
| mode: RenameHandler.RenameMode |
tests/data/test_renamehandler.py
Outdated
| @pytest.mark.parametrize('pattern,repl,mode,files_in,files_out', [ | ||
| (r'file', r'data', RenameHandler.RenameMode.MOVE, | ||
| ['data/data.txt','data1/data1.txt'], | ||
| ['data/data.txt','data1/data1.txt'], | ||
| ), | ||
| (r'(?P<name>data[^/]*.txt)', r'new_dir/\g<name>', RenameHandler.RenameMode.COPY, | ||
| ['data/data.txt','data1/data1.txt'], | ||
| ['data/data.txt','data1/new_dir/data1.txt', 'data/data.txt','data1/new_dir/data1.txt'], | ||
| ), | ||
| (r'data[^/]*/', r'', RenameHandler.RenameMode.SYMLINK, | ||
| ['data/data.txt','data1/data1.txt', 'data2/data1.txt'], | ||
| None, | ||
| ), | ||
| (r'(?P<name>data[^/]*.txt)', r'newdir/\g<name>', RenameHandler.RenameMode.SYMLINK, | ||
| ['data/data.txt','data1/data1.txt', 'data2/data1.txt'], | ||
| ['data/data.txt','data1/data1.txt', 'data2/data1.txt', | ||
| 'data/newdir/data.txt','data1/newdir/data1.txt', | ||
| 'data2/newdir/data1.txt'], | ||
| ), | ||
| (r'data[^/]*/', r'', RenameHandler.RenameMode.MOVE, | ||
| ['data/data.txt','data1/data1.txt', 'data1/data2.txt'], | ||
| ['data.txt', 'data1.txt', 'data2.txt'], | ||
| ), | ||
| (r'data[12]/', r'data/', RenameHandler.RenameMode.MOVE, | ||
| ['data/data.txt','data1/data.txt', 'data1/data2.txt'], | ||
| ['data/data.txt', 'data/data2.txt'], | ||
| ), | ||
| (r'replacement$', r'dummypath', RenameHandler.RenameMode.COPY, | ||
| ['dummypath/somedata.tar.gz','replacement'], | ||
| ['dummypath', 'replacement'], | ||
| ), |
There was a problem hiding this comment.
repl should not be an r-string, right?
Also, if you decide to allow also compiled patterns I would add or convert 1-2 of the parameterisations to use re.compile(r'...') as pattern
There was a problem hiding this comment.
Tried to change this but now it complains about repl strings that contain \g<something>. So I guess that at least some of them should should be r-strings.
There was a problem hiding this comment.
This looks great, and aside from the marked docstrings the design is very neat. I agree with all that was said before and look forward to seeing the actual "pipelines" in action.
Will mark only as comment for now, until all of B.'s points are addressed, but looks good to me otherwise.
johannesbulin
force-pushed
the
johannesbulin.datahandler.#6
branch
from
65dedfc to
854b655
Compare
(See issue #6)
Added support for data pipelines to
ifsbench. The main feature is aDataHandlerbase class that can perform a given action on a working directory. Currently I've implemented:RenameHandler- for moving/renaming/copying files, using regular expressions.NamelistHandler- for modifying namelists.ExtractHandler- for extracting archives.Talking points / request for feedback:
InputFileconcept that we've used before (with file hashes and other things). If you think that this should be included, I'd be happy about some suggestions.RenameHandler.RenameModefor example). It's a bit clunky to write but I haven't come up with a better solution.